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\ ' ^ *Validity 
'IDENTIFIERS *Internal Validity " 

■Abstract ■ ■ ■<. 

• Internal validity is described as a matter" of how 
wdll a\ [particular irfstance of data collect^^n or generation can be 
ileecribed and. explaiined. It is a property o^vthe procedures used in 
^ihe collection or generation of data. The notio^n of internal validity 
fs examined in orderl to establish a method of gnantite^tively 
i^Stimating it. A coefficient of internal validity is^efined by the 
equation that it equails one divided by one more than th« number of 
•iklter native independent plausible hypotheses. If experimented 
Iprocedoxes rule out all alternative plausible hypotheses, then the 
^internal validity coefficient equals one. As the number of 
alternative hypotheses Approaches infinity, the internal validity 
coefficient approaches zero. Alternative hypotheses may be considered 
;tb be equivalent 'to alternative independent variables that are 
discrete and non-lover lapping and cann6t be ruled out and are 
In'ifependently definable. A Suggestion is gifren for weighting 
independent variables according to their Hcai\lc order of being 
i»lausible. This view is coherent with Barman's view of induction as 
inference to the best explana^on. (AuthorA^TM) , 
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Abstract 

The notlop of Interfial validity Is examined for purposes 
of eatabllshing a method of quantitatively estimating it. The 
logical problem of identlfyitig independent alternative hypothesei 
is considered as well as their relative plaualblii^y weighting. 
Finally, the' question of Internal validity is viewed as being 
a form of Ha^on*av!' inference to the best explanation". 



TOWARD A QUANTITATIVE ESTIMATE , . 

OP 

. INTERNAL VALIDITY 

* • . . . ■ 

Analyses of the conclusions about how the results of experl- / 
ments shotild be lnterp>retdd &re comaonly conducted in terms of exter- 
nal and Internal validity/^ The continued elaboratdrtii^ of these ideas 

■ ■ • ■ ■ . 1 

vould» it seems* increase the adequacy of these analyses. • The 
present paper explores tl\e nature of the latter type of validity 

■ ■ . ; . ^ ' I . • 

in dn attempt to further refine this concept within educational research. 
This attempt ^eems to involve (1) rendering ^ore precise our intul- 
tlons or our working notions ^bout internal validity, and (2) marking . 
the klclds of logical problems^ncbuntered in tryikg.to give such . 



an accdunt. 



4r 



.^tuaies are often ordered in terms of internal validity. i)ne 

^ ^ - ■ r ■ ■ ' 
study is Judged as possessing greater internal validity than another, 

indicating that Jthe qoncept is basically a quantitative notion. 

The present paper attempts to develop a method for the quantita-* 

'Pi . 

tive ^^tlmation of internal validity. While there are pitfalls 
involve^ in s'^ch an effort, it is hoped that the effort will stimu-^ 
latc^ an (^interest ia^^he^problem by logic-*mlnded educationists • 
i. The Problem' of Internal Validity 
To keep -the discussion as concrete as possible within the demands 
of purpose, consider the notorious one^ group pretest /post-^test 
design. ^ Assuming that the pre- and post-readings on the dependent ^ 



variable (i) reveal a statistically significant gain on the post- 

-. • ■ ...... ^ 

8core8f the Question becQmes: what produced this gain; or^ how ^ 
can the observed change In the dependent variable best be explained?. 

One readily ^ available candidate, and one In yhlch there will 
be great Interest, Is of course the Independent varlableCs), l«e«9 

the treatSBent(s>. explains the observed difference. Or moi;e accurately 

. ■ . . *\" 

the hypothesis ^ssertlng* a relation between the Independent variable 
and the dependent variable -can serve In the explanation of the 
observed difference. Ther^ are, however, alternative explanations 
of this gain. Th^ weaker the design of the study the more alternatives 
one can find. Intuitively, It seems that the more of these alter- 
natives which are plausible, the lower the Internal validity of the . 
Study. The question^f internal^ v Is taken to be: Did the 

treatment make any difference or have any effect upon the criterion 
variable? If a change in the criterion readings is observed, how 
sound or warranted is the explanation of this difference by the 
research hypothesis? If no change is observed, then did the treat- 
ment hrve an effect which was masked by otlier extraneous factors? 
Xntemai validity is a matter of how well a particul^irxmtdhce of 
data colleotlon or generation can be described and explained. 

From these basic notions, It is seen that the property internal 
v^^Ldity is a property of the procedures ^^used in the collection • ' 
or generation of data on ^ particular occasion. Internal validity 
is a variable for the population of data gathering methods . More^ 
over, it is not taken to be a categorical property for the loglc^ 



In-uae or at' least the language^-ln-use tafers to "the degree of lnt!er»- 
*nal validity"; thus It seems that a reasoaable approach to the 
concept o£ Internal validity Is to view It as a quantitative variable. 

Inj:emal validity Is usually mentioned in the context of exper- 
imental research. /There is talk of the two kinds of experimental 'N^ 
validity. It seems that oat^ could properly speak of the Int^niial 

, validity of ex post facto istudles. One can in these eases ask^ 
did the hypothesized independent variable catise the observed chdEnges 

-in the dei>ehdent variable? Maniptilatlon of the liidependent variable 
allows one to know more about the independent variable thac( do most 
ex post facto studies;* but this amoont^ to one's having a higher 
4jegree of confidence in the description of the variance of the 
independent variable than most ex post facto studies allow. It 

does not« howler, show that th6 question of internal validity, 

• . ■ * ^ '.■■'*■. .... . 

which is a question of how data is to be intexpreted^ is not appro- 
priate to causal comparative work. It seems that the question of 
internal validity is relevant to any methods used to gather data 
relevant to any hypothetical causal-relationship. / 

II. Internal Validity Measure Ftmction 
The problem of Internal validity is erne of how yell certain 
particular events can be explained. As noted above, there will 

be much Interest at the conclusion of data collection in determining 

, -* ^ 

hbw well the research hypothesis can explain the results obtained. 
• , ■ ' ' •■' " ■.. * . ■ ^ 

To test the ground for the* treatment variable explanation, the follow-, 

* ■ * - • ' 
Ing jproceduxe Is suggested. 



Id 



V 



(la) 'Assunui the data reports are correct and that there wae 
a change In the dependent veritable, ((lb) would be the 
no change i case and .will not be considered here though ^ 
the following, applies to it as well.) If the assumpcion 

that .the data reports are* correct cannot b^ madevfor 

If ' » . ■ . ■ ' 

whatever reason, then the question o£ internal validity 
evaporates. There is no question of how to explain 
a single event if the nature of that event is unknown. 

9 

'(2) .Assume that the: treatment variiable did not produce the 

change in the dependent variable. ' 
(3) A&k: what piroduced change? Or, how can this change 

' ■ ft ' HI " 

be explained? 

X4a) If u^on careful examination of the research procedures 

no reasonable or plausible explanations can be ^ound which 
are consistent with (la) and (2), then one Is forced by 
rationality to reject either (la) pr (2). If the truth^ 
lof (la) is not established then this procedure^ unnec- 
essary. The rejection of (2) is the acceptance of the 
research hypotheses, i.e., of the treatmiknt->variable 
explanation. This conclusion is epistemlcally open 

■ ; - A-' • ■ . 

and practically closed^ It is possible that one may 

» 

detect or cdnstruct an explanation of the results at some " 

future time. One cannot know for sure that do- answer 

to (3) is possible. Conclusion (4a) Is open in this sense. 



•■•./ 



ERIC 



with respect to the ptactlce of sdencei^ ho«ever» one 
must vl^ thinl^s .differently. In aclence there la always 
\ / tKe^^oa&iblllty of some further data forcing the relnter- 
pretiatlon of past findings producing new conclusions.' 
The scientist must, nevertheless* show the best conclu«* 
slons he can on the basis of the total evidence available. 
(4b) If one could find explanations of the observed chap^es % 

. ' ■ . . • . ■ ■ ■. ^ 

which are consistent with (la) and (2), then the yeracity 
/ef the tre'atment explanation Is suspect. There ei(lst ' 
cpmpetlng explana^ons of what happened; and there Is 
no way to choose between 'these coiq>etltlg hypotheses within 
the l/imlts of the data of this particular itudy. Inter- 
nal validity should be^ a function of the D|umber andt qual- 
ity of these alternative hypotheses. The greater the 
numbei^ of such, the lower the degree of Internal jralldlty- 
In caafes where no alternative can be found, (4a), lntema;l 
- validity, should be highest, decreasing as the list o|f 
alternatives grows. ^ 
- Consider the following equation, where Is the internal valid- 
Ity and N is i^e numb«§r of alternative hypotheses: 



^1 



N + 1 



This definition of internal validity meets the 'jirequirements of our 
intuitions. It assign's the internal validity unity whfe^ there aye ^ 



no Altarnal|;lv« hypothM.M. As th* iiuiiib«r of alternatives apprdAcHss 

■ L • ' • ■ 

Infinity ^he function value apt>Toafih«8 zero. , ^ 

Several conceptual questions now present themselves. . It -Is 

• * \ I 

obviously critical how one counts alternative hypotheses. Whaf 
is requl^d Is a way of determining thft logical Independence of 
the prpposed 'alternative, hyp^heses. ^econ^y, some may question 
the above function On the grounds that It treats all alternative 
hypotheses as equally well supported; but is it not the case that 
certain alternative hjrpotheses are more plausible than others? 

III. Independent Alternative Hypotheses r 
Let & be the* set of non--ref uted alternative hypotheses for a 
given design. Assume tjhat this seti Is practically complete or 
closed (see above section) . But ^hat 4oes It mean to say that any 

^ ■ ^ ■ ■ / ■ ' . ' ■ 

given hypQtliesls is refutied? Ic of course Is not Intended to siiggest 
that science Is to produce absolute .kncrwledgee To be relevant a 
hypothesis must present an Independent Variable which Is linked 
hypothetlcally to the dependent variables whose measures constitute 
the data; the hypothesis must explain the data at hand. A refuted 
hypothesis which explains the data Is a hypothesis which Is ruled 
out on one of three grounds : ' « / 

(1) other well-accepted hypotheses contradict this explanation. 

(2K This design used to produce these, data rules out thls^ 
- . , explanation of tiie data* 

(3) Our metaphysical assumptions render this explanation * 
••impossible*'. 



(1) Wh«n rtvlmrlng the data, on* will raslst using a hypothesis 

■ , ..■ ■ - • . ,' • ■ *- ^ 

to, axpialn or Intarprat what tiappenf d wl\lch cbntradlots othar wall- 
supported rftlatlonshlps. It la not that ya nayar quaatlon tbat ^htth 
la "aatabll8li^4" but oiay that wa will maka such challangaa only 
a£ter^*W4|av^ tha Intamal validity o£ our daai^na - • 

All raaaurch 8tudi99^1mng--tsi^thar as it vara. (2) Qxia tha^jpr ^ 
purpoaaa of the creation of a rMMrch dasigif ia to rula out altar-^ 
>a$lve after-tke-fact accounts t>f ouK^rasults. Wa plan ah^ad 



regard to being able to drw dafanslbla conclusions. (5) One could 
construct alternative hypotheses as to why the expariment^al grolup 
'did .bettet than tha control group » by referring to entities such 
as demons — there are invisible demons who.'ulke projgraa learning and' 
who always confuse people vfao do not use this meithod, like our 
control group was not taught by the prbgrmi methods Such explan* 

atibns -are too farfetched; but to realize ^ that scientific inquiries 

« ^ .- / * ■ • » 

^do operate out of a basic metaphysical frjUDework^ or "blueprint" 

as Haxvell calls it, is a very important part of our conception of 

the nature of science. See Maxwell (1974)*. ^ 

Given t^t we*^ hicve identified the set (aU non-refuted alter* 

■■•^ . . • - ? ■ ^ ' - ^ / ■ ^- . . ' • 

native hypotheses)^ we encountef the questtion of the tiniquenesa of 

... ... . ^ ' , J* 

this set. sCan^thls list be $iveii in such a fashion that wh^n it is 

coimt^d a stable number can be obtained? Or state^f differently, ^ 

is the iq^bershlp of SL_unlquely describable? What we require Is 

* . - . . . 

a, iDathod of^>q:ltlng a' basic l±st of the members of S. 



, Each navbttr o£ wl»ir hAv« th« aani dttt^^ 
thuMf What w« are aettMLlly aaklng for la a llat of altamatlva Inda- 
pendant -v^rlablas'^alcernatlvea to th^ treatment variable (8)t t. * ^ 
We want a llat of discrete » non-overlapping Independent yarlablea 
irtilch have dot been^ruled out. Juat aa we require the Independent 
and deiS€|ndent varlablee, of any hypotheela to be Independently defin- 



able— if they are npty the hypotheala becpmea true (tfr falae) or parV 
tlally" true <or false) by definition- we require that the Independent 
varlableiB of the' members o^ alee^ Ixfdependently definable! ^ ^ 
In^other words » the deflnlens ^of each Independent varlable(s) of 
the members of S must be mutualj.y distinct. This ^guarantees that 

we are dealing' with discrete alternatives. HOreoyer, It rules out 

' < \ \ ■ ' ' ' ' * 

the posslblUlT that one Independent variable oz the members of ^ 

• ' " ^ ' * ■'■». ' 

will entkil another. The- independent variables aauicclated with 

/ V .-^ / ■ . -\ I ' • /■' . . ' . 

will thus be loglc^^ly basic or atdmic^ 'as it w^re^ . 

/This, appeal to logically discrete definiens^ brief as it is/ ^ 



does ' resolve the f Irs t 



problem: how to .determine the logical 



^ndepen^enbe of the elements of the set of alternative hypotheses 
' to the reseaflrch ^hypoth^sis/, 

IV. Plausibility of Alt^ematlve^ Hypotheses 
^e second problejn mentioned above i^as in effect a iceiectlon of 

• - ^ ' A . / ^ / 

our definition "V^". That runction threats all alternative hypotheses 

. • * • - 1 • . / • 

as if they wete equally merlJLorious;^ but some alternative independent 
variable^ are going to^e moVe plausible than others. In sfn&e 



taper jjBUtal ilt^tidna, t«stlag will bft a b«ttttr 4>«t than mftturftr 

tlon> ytt this sttms to b% Ignored by the •IsqplMlnded {unction ^Iven 

. " ' • ■ . *^ • , \ ' , 

eboye* _ " - • • 

' ^ ' ^ /I J ■ * ' ' 

It wonid-be an Improvement to rsnl^ eltemetJ^e hypouieeee or ^ 

Independent verlebltfs Including the treatment variable In thla 

I * * ' * . ' . ; ' • ^ 

llat at tha appropriate rank. Thla ^at vi.U h*va N + 1 manbara ^ 

> ■ * ■ _ ' ' " . ^ ,• • . • ■• ^ 

alnce N la tha nunibar of altamatlva hypothaala with raapact t6 

t. Aaalgn'tha weight, of 1{ •>• 1 to tha ^r at member of the lla't,^ 

• , \j — • • . ■ ... 

(N 4* 1) - 1. to the second* and so forth. Sua the^ weights. Intenial 

validity can be defined as follows: j * ■ j 

; ■■ tr - Welttht: of! t - . ' ■ ^ | 

Sum of^wclljghta • • • f' > 

Five situations are preseht^ed be.low, [together with their gener'-L 



■J 



a'ted Interhal validity measures^for llliutratlve purpoaea'f 



J- 
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1/6 



This function ror obtalnlfig Values for entails the fact that 
can never fall within the open Interval (1, 2/3).; If one wanted 



Vi values higher than 2/3 but less than uhlty, one would WSvfe Co 



move away from rank ordering and into some type of Interval scale. 
This, of course, wotild further strain our notions of plausibility 

with regard to alternative hypotheses; it would }i'e more dlf 51cult 

■ ^ ■ • ' , ' . ■ 

to. make the required weighting asslgxunents . ^ 

/ On the positive, side, I think that this method, elemeatiary as 

it is, would help us gain a sense of relative significance when - 

we are 'trying to summarize several research studies on a given topic. 

Probably many people have a sense of this anyway— but one of the 

functions of logic is to explicate our Intuitions. 

V. Inductive Issues 

f - ■ ■ ■■ • , 

The classical or NeymSan-Pearsoi^ statistical theory does not, of 
course,, assign probabilities to hypotheses* Thus, some might wonder 
whether the proposal offered here is aesthetically coiiq)atlble with 
this theory.- should not be seen as a probability; rather it is 
a Weight of _t against its alternatives. This kind of weighting 



is indigenous to classical statistics. „ " 

Both ^actual practice ani^ the proposal of this paper fall 
nicely liito Barman's view of Induction as "inference to the best 
£ cplanatlorf^ (1965). 



In making this inference onje Infers, £i:om the fact that' 
a certain hypothesis would explain^ the evidence, to the 
truth of that hypothesis. \ln general, there will be 
\ /several hypotheses which might explain the evidence, 
\ so one must be able to reject all stich alternative hypoth- 
^\esis before one is warranted in making the inference. . 
Thus, one infeps, f^om tjie premise that' a given hypdthesis 
woul4' providk a '^better" explanation for the evidence than 
wo^id^ai^y other hypothesis, to the conclusion that the 
given hypothesis is true (p. 89). 



I think that it is evident that Harman's inference to the best 
exiplanation is precisely the kind of Inference at issue in the 

questions of internal validity. Moreover^ we are considering the 

■ • ' '' ' ' ' ' ■ 

cases where we cannot "rejecfvall sutch alternative hypotheses." 

^- ' . . . • - r • 1 

Before we go to the public with claims of efficacy we will want i 
ta be ab^ to reject all altexrnatives; but as we talk to each other 
we rei|uire some way of estimating the relative merits of a/temative 
-y^aya of viewing what happened in various studies. The aot:ount of 
estimating internal validity developed' in . this paper is proposed 
^as such a way* 

Finally y there Is an Interesting aspect of Harman's no.tion as 
it relates , to an issue in the logic of scietnce. Some philosophers 
believe that Harman's view is circular in th^t iq^duction' to the best 
explanation presupposes a way of determining "best" which is itself 
^n inductive -process « However, I will risk the following claim: 
While Inference '^to the best explanation is shaky las a method of ^ 
'producing general scienti^c knowled^is, it is' both the method- * ^ 
in-use for determining internal validity .and la a sound move for 
so doing, when viewed logically.. W^hln the confines of a single 
experiment, i.e., where there is no concern for generalizabllity >- 
thie only rationally defensible way of inferring "what happened'' 
is the method /ieacribed by Harman. 



